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INTRODUCTION 


This  paper  is  intended  to  provide  an  overview  of  the  Software  Process  Improvement 
(SPI)  efforts  of  Naval  Air  Systems  Command’s  (NAVAIR)  Air  4.0,  Research  and 
Engineering  Competency.  NAVAIR  4.0  provides  life-cycle  systems  development  along 
with  operations  and  maintenance  support  for  the  aircraft  and  weapons  of  the  U.S.  Navy 
and  Marine  Corps.  NAVAIR  4.0  is  distributed  across  the  country:  Patuxent  River, 
Maryland;  Lakehurst,  New  Jersey;  Orlando,  Florida;  and  China  Lake  and  Point  Mugu, 
California  (Figure  1).  While  there  are  NAVAIR  facilities  located  in  Italy  and  Japan,  this 
report  will  focus  on  the  24  discrete  software  engineering  teams  located  within  the  U.S., 
and  specifically  on  the  SPI  efforts  of  the  six  teams  that  were  early  SPI  adopters. 


FIGURE  1.  The  Locations  of  NAVAIR  Facilities  Within  the  U.S. 


The  early  SPI  adopters  mentioned  above  were  the  software  development  teams 
within  the  AV-8B  Joint  System  Support  Activity  (JSSA);  the  E-2C,  EA-6B,  P-3C,  and 
Tactical  Aircraft  Electronic  Warfare  (TACAIR  EW)  Software  Support  Activities  (SSAs); 
and  the  F/A-18  Software  Development  Task  Team  (SWDTT).  These  teams  ranged  in  size 
from  less  than  10  to  more  than  70  NAVAIR  software  engineers  and  support  contractors. 
They  were  organized  under  the  Director  of  the  Engineering  Division  of  the  Research  and 
Engineering  Group,  Code  4.0,  of  NAVAIR  Weapons  Division  (WD). 
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BACKGROUND 


Over  the  last  several  decades  NAVAIR,  the  parent  organization  of  Code  4.0, 
Research  and  Engineering  Group,  has  experienced  tightening  budgets,  decreasing  labor 
pools  (Figure  2),  increasing  software  complexity  (Figure  3),  and,  finally,  the  demands  of 
the  Global  War  on  Terrorism  (GWT).  Throughout  this  period,  NAVAIR  has  worked  to 
meet  the  challenge  of  accomplishing  its  mission  while  procuring  the  new  aircraft 
necessary  to  meet  its  future  obligations  to  the  Fleet. 


FIGURE  2.  Manning  History  and  Forecast. 
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Increasing  complexity  has  increased  need  and  demand  for 
development  discipline  and  integration  skills 
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FIGURE  3.  Aviation  Software  Complexity. 


To  meet  this  challenge,  process  improvement  efforts  were  initiated  throughout 
NAVAIR,  in  all  business  areas,  including  administration,  contracting,  support,  and 
software  development.  These  initiatives  began  to  take  shape  for  Code  4.0  in  1998  when 
AV-8B  joined  F/A-1 8  in  the  pursuit  of  process  improvement.  Between  April  and 
September  2002,  NAVAIR  issued  a  set  of  five  formal  instructions  as  guidance  on  process 
improvement  for  software  acquisition,  development,  and  life  cycle  maintenance  (see 
Appendix  A).  One  of  these  instructions,  NAVAIRINST  5234.2,  was  based  in  part  on 
Code  4.0’s  research  into  process  improvement  tools  and  techniques  (Reference  1,  page 
9).  In  December  2002  NAVAIR  4.0’s  voluntary  effort  was  bolstered  by  the  passage  of 
the  U.S.  Federal  Government  statute,  Public  Law  107-314,  the  Bob  Stump  National 
Defense  Authorization  Act  for  Fiscal  Year  2003.  Section  804  specifies  that  software 
acquisition  programs  must  meet  the  following  requirements: 

•  Shall  have  a  documented  process  for  planning,  requirements  development  and 
management,  project  management  and  oversight,  and  risk  management. 

•  Shall  have  a  metrics  for  performance  measurement  and  continual  process 
improvement. 


5 


NAWCWD  TP  8642 


•  Shall  have  a  process  to  ensure  adherence  to  established  process,  and  requirements 
related  to  software  acquisition. 

In  this  environment,  the  goals  of  the  SSAs  were  to  improve  the  maturity  of  the 
software  development  processes,  to  realize  cost  savings,  and  to  deliver  higher  quality 
products  to  the  Fleet — in  essence,  to  meet  the  challenges  of  their  missions  while  at  the 
same  time  meeting  NAV AIR’s  stated  organizational  goals,  as  stated  in  2005  (see 
Appendix  B). 


THE  SOFTWARE  PROCESS  IMPROVEMENT  TOOL  SET 


The  initial  SPI  efforts  of  the  individual  SSAs  were  not  coordinated  across  NAVAIR 
4.0.  Each  SSA  acted  as  an  independent  entity  within  the  overall  effort,  starting  at 
different  times  and  selecting  SPI  tool  sets  specific  to  the  needs  of  their  individual 
organizations  (Table  1). 

TABLE  1.  The  SPI  Tool  Sets  of  the  NAVAIR  Early  Adopters. 


Organization 

Process  improvement  tools 

CMM 

CMMI 

TSP 

EVMS 

HPO 

TSPm 

AV-8B 

V 

S 

V 

E-2 

S 

V 

EA-6B 

V 

V 

P-3C 

S 

S 

V 

V 

TACAIR  EW 

S 

y 

V 

F/A-18  SWDTT 

S 

V 

s 

CAPABILITY  MATURITY  MODEL  (CMM) 

A  Carnegie  Mellon  University,  Software  Engineering  Institute  (SEI)  framework  for 
incremental  process  improvement,  CMM  consists  of  best  practices  that  cover  the 
complete  product  life  cycle,  starting  with  defining  requirements  and  continuing  on 
through  maintenance  of  the  delivered  products.  The  CMM  is  broken  into  five  levels  of 
maturity,  each  with  a  discrete  set  of  practices  that  characterize  an  organization  operating 
at  that  level.  It  is  a  useful  tool  for  appraising  organizational  maturity  and  for  guiding 
incremental  process  improvement  efforts.  CMM  is  sometimes  used  interchangeably  with 
Software-CMM  (SW-CMM). 
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CAPABILITY  MATURITY  MODEL  INTEGRATION  (CMMI) 

This  is  SEI’s  follow-on  model,  which  replaces  several  process  improvement  tools, 
including  the  CMM.  CMM  is  the  baseline  for  the  CMMI,  but  expanded  to  include 
System  Engineering  (SE)  and  generalized  to  accommodate  a  wider  variety  of  business 
models.  It  encourages  organizations  to  focus  process  improvement  efforts  based  on  one 
or  more  specific  areas  of  their  business,  instead  of  requiring  one  all-encompassing 
process  improvement  effort.  In  this  way,  organizations  may  pursue  process  improvement 
in  only  those  areas  they  deem  most  urgently  in  need  of  process  improvement. 


EARNED  VALUE  MANAGEMENT  SYSTEM  (EVMS) 

This  is  a  tool  for  program  management  that  allows  visibility  into  both  technical 
issues  and  cost  and  schedule  progress  on  projects  and  contracts.  Analysis  using  EVMS 
can  provide  an  early  warning  for  issues  with  a  project  or  contract,  from  as  early  as  the 
point  at  which  15%  of  that  effort  has  been  completed.  In  order  to  use  EVMS,  program 
management  must  ensure  that  their  effort  is  fully  defined  from  the  beginning,  including  a 
bottom-up  plan.  In  this  plan,  each  discrete  task  will  have  an  associated  value  that 
corresponds  to  a  percentage  of  the  total  work  effort.  This  will  allow  measurement  of  the 
bottom-up  plan  for  the  entire  period  of  performance.  The  data  collected  provide  a  way  to 
show  actual  performance  improvement,  and  become  the  basis  of  modeling  predictable 
performance  for  future  projects.  EVMS  is  one  of  the  CMMI’s  best  practices  for  project 
planning. 


HIGH  PERFORMANCE  ORGANIZATION  (HPO) 

HPO  is  an  organizational  improvement  plan  that  is  guided  by  the  Diagnostic/Change 
Model  for  Building  High-Performance  Organizations  as  developed  by  the  Common¬ 
wealth  Centers  for  High  Performance  Organizations  (CCHPO).  It  is  commonly  referred 
to  as  the  Diagnostic/Change  Model.  The  work  is  typically  initiated  with  a  workshop 
(“Teamway”)  designed  to  help  a  group  develop  the  skills  required  to  improve  their 
performance  by  continually  using  the  Diagnostic/Change  Model.  Conducted  with  intact 
work  groups,  the  3-  to  5-day  workshop  is  tailored  to  each  group’s  needs,  concerns,  and 
issues. 


PERSONAL  SOFTWARE  PROCESS  (PSP) 

The  PSP  is  a  SEI  methodology  for  developing  high  quality  software.  It  is  based  on 
the  practices  that  are  characteristic  of  CMM  and  CMMI  Maturity  Level  5  organizations. 
It  uses  standards,  methods,  scripts,  measures,  and  forms  to  provide  a  highly  structured 
and  disciplined  framework  for  individual  software  developers  to  use  in  their  daily 
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software  development  efforts.  The  measurements  collected  are  used  to  improve  the 
quality  of  the  products  developed. 


TEAM  SOFTWARE  PROCESS  (TSP) 

The  TSP  is  based  on  the  concepts  and  practices  of  the  PSP  and  is  the  methodology 
through  which  PSP  may  be  applied  to  team-based,  software  development  efforts.  All 
software  engineers  participating  in  a  TSP  team  are  required  to  be  PSP  trained.  TSP  and 
CMMI  are  complementary,  and  they  work  best  when  introduced  into  an  organization  at 
the  same  time  (Reference  1,  page  5).  The  data  collected  from  these  teams  are  used  to  set 
performance  and  quality  objectives  for  the  organization. 


TEAM  SOFTWARE  PROCESS  FOR  MULTIPLE  TEAMS  (TSPm) 

TSPm  is  an  SEI  prototype  methodology  derived  from  the  TSP  that  is  intended  to 
facilitate  the  application  of  TSP  principles  in  situations  where  there  are  multiple  TSP 
teams  engaged  in  developing  sub-units  of  software  for  a  common  product. 


THE  SPI  JOURNEYS  OF  THE  EARLY  ADOPTERS 

AV-8B  JOINT  SYSTEM/SOFTWARE  SUPPORT  ACTIVITY 

The  AV-8B  JSSA  started  its  SPI  process  in  1998.  At  that  time,  AV-8B’s  active 
project  (“Project  A”),  was  estimated  to  be  9  months  behind  schedule  (a  17.6%  schedule 
overrun)  and  $49  million  over  budget  (a  28.3%  cost  overrun).  Determined  to  address  the 
root  causes  of  this  disappointing  performance,  AV-8B  created  an  independent  review 
team  to  inspect  the  Project  A  software  development  effort.  The  team  completed  the 
review  and  recommended  that  AV-8B  pursue  process  improvement.  To  focus  that 
pursuit,  AV-8B  determined  that  their  top-level  SPI  goals  would  be  to  implement  EVMS 
for  tracking  project  cost  and  schedule,  and  to  implement  HPO  concepts  in  order  to 
develop  a  more  mature  software  development  process.  The  plan  also  called  for  obtaining 
an  EVMS  certification.  The  Department  of  Defense  (DOD)  criteria  for  obtaining 
certification  are  listed  in  Appendix  C. 

AV-8B’s  progress  was  swift.  EVMS  training  began  in  October  1998  and  AV-8B’s 
Software  Engineering  Process  Group  (SEPGsm)  was  formed  in  March  2000.  This  was 
followed  by  TSP  training  in  October  2000  and  HPO  training  in  January  2001.  By 
May  2001  AV-8B  had  been  assessed  at  CMM  Level  2  and  by  October  2001  they  received 


smSEPG  is  a  service  mark  of  Carnegie  Mellon  University. 
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their  DOD  EVMS  certification,  the  second  organization  in  the  Federal  Government  to  be 
certified.  In  September  2002,  AV-8B  commissioned  a  SW-CMM  CMM-Based  Appraisal 
for  Internal  Process  Improvement  (CBA  IPI).  The  assessment  concluded  that  AV-8B  had 
achieved  SEI  CMM  Level  4.  Given  a  start  date  of  March  2000  for  CMM,  AV-8B 
managed  to  reach  CMM  Level  4  in  2 Vi  years.  (The  SEI  average  for  progressing  from 
CMM  Level  1  to  Level  4  is  5!4  years  (Reference  2,  pages  3-4).) 

The  AV-8B  team  attributed  their  rapid  advancement  through  the  CMM  levels  to  the 
use  of  the  TSP  and  a  culture  of  process  improvement.  TSP  provides  a  quickly 
implemented,  flexible  process  framework.  The  guidance  contained  in  the  SEI  Technical 
Report,  Relating  the  Team  Software  Process  to  the  SW-CMM  (Reference  3),  helped  the 
AV-8B  SEPG  focus  and  prioritize  its  efforts.  The  technical  report  proved  a  valuable  tool 
in  applying  process  improvement  techniques  in  the  most  efficient  manner,  shortening 
what  might  otherwise  have  been  a  long  learning  curve.  Furthermore,  TSP  distributed  the 
process  improvement  responsibilities  across  the  project  teams,  so  that  process  changes 
originated  from  within  the  development  teams.  This  increased  the  entire  AV-8B  team’s 
commitment  to  those  changes.  The  AV-8B  JSSA’s  Leader,  Dwayne  Heinsma,  added 
“The  recipe  for  accelerating  AV-8B’s  climb  up  the  software  maturity  ladder  centered 
around  identifying  champions  and  using  process  discipline  as  an  enabler.  These 
champions  included  a  Personal  Software  Process/Team  Software  Process  champion 
leading  the  software  team;  an  organizational  process  champion  leading  the  development 
and  the  institutionalization  of  organizational  standards;  senior  managers  championing  the 
overall  effort  and  removing  roadblocks  (establishing  both  TSP  and  an  Earned  Value 
Management  as  the  standard  way  of  doing  business  at  the  JSSA);  and,  most  importantly, 
an  excellent  team  of  software  engineers,  systems  engineers,  and  product  integrity  support 
members  that  made  it  all  happen.” 

TSP  and  EVMS  improved  AV-8B’s  cost  and  schedule  performance  as  well.  Once 
EVMS  was  put  into  use,  schedule  and  cost  variances  were  brought  down  to  within  10%; 
the  introduction  of  TSP  brought  them  even  lower  (Table  2). 

TABLE  2.  AV-8B  Schedule  and  Cost  Variances  Related  to  EVMS  and  TSP. 


Project 

Date 

Schedule  variance 

Cost  variance 

Used 

EVMS? 

Used 

TSP? 

Project  A 

At  7/98 

17.6%  overrun 

28.3%  overrun 

No 

No 

Project  B 

Complete  4/02 

50.0%  overrun 

300.0%  overrun 

No 

No 

Project  C 

Complete  5/04 

5.0%  overrun 

8.1%  overrun 

Yes 

No 

Project  D 

As  of  7/04 

0.5%  overrun 

1.5%  overrun 

Yes 

Yes 

Project  E 

As  of  5/04 

1.1%  overrun 

6.9%  overrun 

Yes 

Yes 

AV-8B’s  excellent  record  in  cost  and  schedule  estimation  continues  to  this  day. 
Figure  4  is  an  Earned  Value  Chart  for  AV-8B’s  Project  F  Mission  Systems  Computer 
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(MSC)  Project  Software  Development  Team.  The  chart  was  generated  by  the  same  TSP 
tool  that  the  team  uses  to  enter  and  track  their  project  plan  and  performance  data.  As  can 
be  seen  in  Figure  4,  after  a  short  delay  at  the  start,  the  project  is  now  on  track  with  their 
original  plan. 


FIGURE  4.  Earned  Value  Chart  From  the  Project  F  MSC  Software  Team. 


The  TSP,  while  also  helping  to  drive  down  schedule  variances,  was  instrumental  in 
bringing  about  a  significant  reduction  in  the  defect  density  of  the  final  software  products 
(see  Table  3).  A  48%  decrease  in  defect  density,  measured  in  defects  per  1,000  lines  of 
code,  occurred  between  two  projects,  B  and  D.  The  same  software  engineers  were 
responsible  for  both  development  efforts,  but  Project  B  used  neither  EVMS  nor  TSP, 
while  Project  D  used  both  (Table  2).  In  another  illustration  of  the  improvement  in  quality, 
a  21%  reduction  in  defect  density  occurred  between  Projects  C  and  D.  While  both  of 
these  projects  were  using  EVMS  to  manage  their  costs  and  schedule,  only  Project  D  used 
TSP. 
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TABLE  3.  AV-8B  Defect  Densities  Related  to  TSP. 


S/W  development 
projects 

Date  completed 

S/W  defects 
during  V&V 

KSLOC 

S/W  defects 
per  KSLOC 

Used 

TSP? 

Project  B 

4/02 

36 

32 

1.13 

No 

Project  C 

5/04 

66 

89 

0.74 

No 

Project  D 

7/04 

260 

443 

0.59 

Yes 

S/W  maintenance 
projects 

Date  completed 

STR  defects 
during  system 
test 

STRs 

resolved 

STR  defects 
per  10  STRs 
resolved 

Used 

TSP? 

Project  E  S/W  Cycle  1 

3/04 

10 

88 

1.13 

Yes 

Project  G  S/W  Cycle  1 

9/04 

2 

40 

0.50 

Yes 

To  calculate  their  return  on  investment  (ROI)  for  TSP,  AV-8B  compared  the  defect 
data  from  two  projects:  B  and  D.  Project  B  was  a  pre-TSP  project  that  had  a  defect 
density  of  1.13  defects  per  KSLOC.  Project  D  was  the  first  TSP  project  and  it  had  a 
defect  density  of  0.59  defects  per  KSLOC.  Table  4  shows  the  ROI  for  TSP  from  savings 
derived  from  the  avoidance  of  rework.  A  hypothetical  cost  for  Project  D  without  TSP  is 
calculated  to  give  an  indication  of  what  the  cost  could  have  been. 

TABLE  4.  AV-8B  Return  on  Investment  in  TSP  After  One  Project. 


Product  size 
(KSLOC) 

Defect  density 
(defects/KSLOC) 

Number 

of  defects 

Cost  of 
addressing 
defect 

Total  cost  for 
addressing  all 
defects 

Pre-TSP  performance  baseline 

Project  B  (pre-TSP) 

1.13 

Hypot 

letical  cost  of  addressing  defects  for  a  non-TSP  Project  E 

Hypothetical 

Project  D  cost 

443 

1.13 

501 

$8,330 

$4,169,831 

Actual  cost  of  addressing  defects  for  the  Project  D  TSP 

Project  D  (TSP) 

443 

0.59 

261 

$8,330 

$2,177,169 

Cost  savings  from  the  avoidance  of  rework 


Cost  savings  from  reduced  defect  density 

$1,992,662 

AV-8B’s  cost  of  TSP  training  and  support 

$225,300 

ROI  from  cost  savings  from  the  avoidance  of  rework 

$1,767,362 

AV-8B  saved  $1,992  million  through  the  avoidance  of  rework.  Even  subtracting 
AV-8B’s  expense  for  initiating  TSP,  the  investment  was  more  than  returned. 
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As  this  document  is  being  written,  AV-8B  is  expanding  from  SPI  for  their  software 
teams  to  Process  Improvement  (PI)  for  the  entire  organization.  They  are  accomplishing 
this  via  a  pilot  project,  Team  Process  Integration  (TPI),  based  on  the  TSP  that  has  been 
modified  to  apply  to  teams  in  disciplines  other  than  software  engineering.  TPI  was 
developed  by  SEI  and  NAVAIR,  and  AV-8B  is  working  closely  with  SEI  to  ensure  that 
the  pilot  project  is  a  success. 


E-2C  SOFTWARE  SUPPORT  ACTIVITY 

The  E-2C  project  office  in  Patuxent,  Maryland,  initiated  its  SPI  effort  in  2000  with 
an  initial  goal  of  “providing  the  Fleet  with  quality  products  that  are  both  affordable  and 
available  when  they  are  needed.”  E-2C  intended  to  achieve  an  SEI  CMM  Level  2  rating 
over  the  course  of  a  6-  to  12-month  project.  After  reviewing  the  available  SPI  tools,  the 
team  adopted  TSP.  It  was  decided  that  the  use  of  TSP  would  commence  with  the  start  of 
the  next  major  project.  During  the  planning  for  that  project  the  scope  of  work  proved  to 
be  larger  than  the  current  work  force  could  handle.  In  2002,  while  searching  for  a 
solution  to  this  challenge,  E-2C  discovered  an  opportunity. 

In  2001,  the  F-14D  SSA  at  Point  Mugu,  California,  was  using  SEI  CMM  and  had 
begun  training  for  (and  using)  TSP  in  its  final  major  software  release  to  the  Fleet. 
Although  the  program  was  scheduled  to  be  phased  out  after  the  completion  of  that 
project,  F-14D  management  considered  TSP  training  to  be  an  excellent  investment  to 
make  in  the  project  engineers.  They  set  the  goal  of  achieving  a  CMM  Level  2  rating  and 
proceeded.  E-2C  learned  that  the  F-14D  program  would  be  closed  in  mid  2003,  and  that 
the  project  engineers  would  become  available  for  work  elsewhere.  The  F-14D  engineers 
had  the  training  and  disciplined  software  processes  that  E-2C  was  seeking,  so  E-2C 
approached  F-14D  management  with  the  idea  of  folding  those  engineers  into  the  E-2C  at 
the  conclusion  of  the  F-14D  program.  The  F-14D  managers  agreed. 

E-2C  launched  its  first  TSP  project  in  May  2003  and  was  formally  assessed  at  CMM 
Level  2  in  June.  In  July  2003  it  incorporated  the  F-14D  engineers  into  E-2C  and  launched 
a  second  TSP  project  at  Point  Mugu.  After  making  some  progress  on  the  project,  E-2C 
re-launched,  replacing  TSP  with  TSPm.  E-2C  found  that  TSPm  was  effective  for 
organizing  and  managing  both  large  and  distributed  teams.  It  was  also  effective  in 
bringing  together  groups  with  different  backgrounds,  by  giving  them  a  common  language 
and  process.  Table  5  shows  the  performance  of  three  of  E-2C’s  early  TSP  projects. 

E-2C  is  currently  transitioning  from  CMM  to  CMMI. 
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TABLE  5.  Schedule  Performance  of  Three  Early  E-2C  TSP  Projects. 


Project  name 

Planned  length, 
weeks 

Actual  length, 
weeks 

Schedule  variance 

SCS-04  ACIS 

33 

32 

3%  under 

SCS-04  MC 

28 

38 

36%  overrun 

SCS-05  SIAP  Phase  I 

29 

34 

17%  overrun 

EA-6B  SOFTWARE  SUPPORT  ACTIVITY 

The  EA-6B  SSA  employs  over  200  people  in  support  of  the  development, 
enhancement,  and  maintenance  of  almost  8  million  source  lines  of  code  (SLOC).  EA-6B 
began  its  SPI  process  in  October  2000  with  the  acquisition  of  Pragma’s  processMax® 
software.  A  process  improvement  lead  was  assigned  in  May  2001  and  a  Process  Steering 
Committee  was  established  in  September  2001.  EA-6B’s  SPI  toolset  included  the  CMM 
and  HPO,  but  the  initial  focus  was  the  implementation  of  an  organizational  level  process 
via  the  processMax  software  tool.  Mini-assessments  were  conducted  in  May  and  October 
2003  and  in  February  2004.  These  preparations  paid  off  when  the  EA-6B  SSA  achieved 
CMM  Level  3  in  its  first  official  appraisal  in  September  2004. 

One  of  the  most  significant  payoffs  for  the  EA-6B  SSA  occurred  during  the  EA-6B 
ICAP  III  Block  2  project.  The  payoff  was  a  substantial  reduction  in  the  number  of 
Operational  Flight  Program  (OFP)  defects  discovered  per  100  system  test  hours.  The  rate 
of  discovery  of  defects  is  a  standard  NAVAIR  product  maturity  measure  used  in 
Operational  Test  Readiness  Reviews  (OTRRs).  The  goal  of  the  EA-6B  ICAP  III  Block  2 
project  was  to  have  a  rate  of  discovery  of  no  more  than  12  defects  per  100  hours. 
However  in  the  last  quarter  of  2004,  EA-6B  noted  that  their  OFP  defect  discovery  rate 
went  from  the  desired  rate  of  12  to  more  than  20  per  100  hours.  Their  ICAP  III  team 
reviewed  the  data  and  in  early  2005  used  the  results  of  their  analysis  to  modify  their 
software  peer  review  process  to  enhance  its  effectiveness.  The  changes  that  the  ICAP  III 
team  made  allowed  them  to  discover  and  correct  a  greater  number  of  defects  prior  to 
releasing  the  next  OFP  Build.  The  result  was  a  reduction  in  the  rate  of  discovery  to  6  per 
100  hours,  surpassing  their  original  quality  goal. 

The  EA-6B  SSA  was  also  able  to  deliver  software  intensive  products  ahead  of 
schedule.  SPI  helped  the  Airborne  Electronic  Attack  (AEA)  Unique  Planning  Component 
(UPC)  project  maintain  and  surpass  their  planned  software  development  and  delivery 
schedule.  Maintaining  these  delivery  commitments  was  critical  to  the  success  of  the 
prime  contractor  development  activities. 
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A  number  of  other  SPI  initiatives  resulted  in  cost  and  schedule  savings  for  the 
EA-6B  Weapon  System  Support  Laboratory  (WSSL).  When  added  together,  these 
additional  annual  savings  come  to  1,231  labor  hours  (two-thirds  of  a  work-year).  These 
initiatives  include: 

•  Upgrading  the  WSSL  Discrepancy  Reporting  (WDR)  process  to  be  CMMI 
compliant  and  utilizing  Lean  Six  Sigma  concepts  to  reduce  work-in-progress.  The 
new  process  provided  web-based  access  for  submission  and  tracking  of  WDRs.  This 
resulted  in  a  reduction  of  manual  labor  for  input/updates  from  4  hours  per  week  to  1 
hour,  an  estimated  annual  savings  of  156  hours.  Metrics  reporting  is  now  automated 
rather  than  manual.  A  new  process  for  testing  and  closing  WDRs  reduced  work-in- 
progress  by  50%  in  the  first  year,  an  estimated  savings  in  labor  of  650  hours  per 
year. 

•  Documenting  and  improving  the  laboratory  engineering  drawing  and  simulation 
Configuration  Management  (CM)  process  to  be  CMMI  compliant.  The  estimated 
savings  in  labor  was  425  hours  per  year. 


P-3C  SOFTWARE  SUPPORT  ACTIVITY 

P-3C  began  its  process  improvement  initiative  in  April  2001  with  the  formation  of 
an  HPO  leadership  team.  Their  initial  goal  was  to  implement  HPO  concepts  within  the 
organization  to  develop  more  mature  software  development  processes.  P-3C  then  decided 
to  add  the  CMMI  and  the  TSP  to  its  tool  set.  A  SEPG  was  formed  in  February  2002  and 
the  first  TSP  launch  was  conducted  in  May  2002.  In  March  2003,  after  perfonning  a 
comprehensive  gap  analysis,  the  organization  transitioned  from  CMMI  to  the  CMM. 
While  the  value  of  CMMI  was  recognized,  CMM  would  allow  a  quicker  pace  (with  an 
earlier  “win”  providing  encouragement  to  the  team).  In  May  2004,  within  27  months  of 
forming  their  SEPG,  P-3C  achieved  CMM  Level  4.  As  with  AV-8B,  P-3C  achieved  this 
in  less  than  half  of  the  514  years  normally  expected  (Reference  2,  pages  3-4). 

In  August  2004  P-3C  perfonned  a  CMMI  gap  analysis  and  transitioned  from  CMM 
back  to  the  CMMI.  In  October  2005,  17  months  after  achieving  CMM  Level  4,  P-3C 
completed  a  Standard  CMMI  Assessment  Method  for  Process  Improvement  (SCAMPI)sm 
B.  While  a  SCAMPI  B  does  not  result  in  an  official  CMMI  rating,  the  results  of  the 
SCAMPI  B  indicated  that  P-3C  was  operating  close  to  CMMI-Software  Level  5. 
Appendix  D  contains  a  definition  and  general  description  of  the  different  types  of 
SCAMPIs.  One  interesting  finding  from  the  P-3C  SCAMPI  B  Appraisal  Findings  Report 
was  that  “TSP/PSP  implementation  has  provided  a  rich  data  source  upon  which  to  build, 
compare,  and  begin  statistical  management  of  selected  processes”  (Reference  4). 


smSCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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P-3C  had  progressed  rapidly  through  the  CMM  levels,  but  what  sort  of  return  on 
investment  did  they  see  for  TSP?  Assuming  all  other  things  to  be  equal,  it  is  useful  to 
focus  on  the  savings  realized  through  the  increased  quality  that  TSP  brings  (i.e.,  savings 
from  the  avoidance  of  rework).  To  do  that,  two  projects  were  compared:  a  P-3C  non-TSP 
project  (for  the  performance  baseline)  and  P-3C’s  first  TSP  project.  Table  6  lists  some 
basic  performance  data  for  these  projects.  It  includes  a  hypothetical  cost  for  the  TSP 
project  based  on  the  non-TSP  defect  density.  This  will  give  an  indication  of  what  it  might 
have  cost  the  project  to  repair  defects  in  “Unit”  test  had  it  not  been  a  TSP  project.  In  this 
example,  P-3C  refers  to  defects  as  Software  Problem  Reports  (SPRs). 

TABLE  6.  P-3C  Performance  Data  Comparison  for  Non-TSP  and  TSP  Projects. 


KSLOC 

Defect 

density/ 

KSLOC 

Number  of 

Average 

Total  SPR  fix 

added/changed 

SPRs 

SPR  fix  cost 

cost 

Pre-TSP  performance  baseline 


Non-TSP 

Project 

27.8 

4.6 

128 

$8,432 

$1,078,284 

Hypothetical  cost  of  addressing  del 

'ects  had  TSP  project  not  used  TSP 

Hypothetica 

1  Project 

38.3 

4.6 

176 

$8,432 

$1,484,032 

Actual  cost  of  addressing  defects  for  the  TSP  project 

TSP  Project 

38.3 

0.6 

23 

$8,432 

$193,936 

Cost  savings  from  the  avoidance  of  rework 


Cost  savings  from  reduced  defect  density 

$1,290,096 

P-3C  cost  of  TSP  training  and  support 

$311,247 

ROI  from  cost  savings  from  the  avoidance  of  rework 

$978,849 

The  number  of  SPRs  generated  during  Unit  testing  of  the  TSP  project  was  seven 
times  lower  than  the  non-TSP  project  because  the  developers  were  able  to  identify 
defects  early  in  the  development  process,  rather  than  having  to  “test”  them  out.  As  a 
result,  the  total  cost  of  removing  the  Unit  test  defects  was  significantly  lower  than  the 
non-TSP  project,  even  though  that  project  was  actually  larger  in  terms  of  total  KSLOC. 

When  the  Unit  test  defect  removal  cost  of  the  TSP  project  is  compared  to  what  that 
cost  might  have  been  had  TSP  not  been  used,  the  savings  by  avoiding  rework  were  nearly 
$1.3  million.  P-3C  invested  $311,247  into  the  training,  setup,  and  support  for  TSP. 
Subtracting  those  costs  from  the  savings  gives  an  ROI  of  $978,849.  P-3C’s  investment  in 
TSP  was  more  than  returned  with  their  first  TSP  project. 
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TACAIR  EW  SOFTWARE  SUPPORT  TEAM 

TACAIR  SSA  provides  post-deployment,  mission  critical  software  and  systems 
engineering,  integration,  and  testing  for  TACAIR  Strike  and  Assault  aircraft.  TACAIR 
SSA  began  pursuing  SPI  in  March  1999.  A  process  guide  was  developed  and  published 
in  October  2000  and  several  HPO  sessions  were  held.  In  August  2002  TACAIR  SSA  held 
a  fonnal  assessment  and,  due  to  some  minor  deficiencies  in  the  area  of  software  quality 
assurance,  just  missed  obtaining  a  CMM  Level  2  rating.  Applying  the  lessons  from  that 
assessment,  by  August  2004  TACAIR  SSA  was  assessed  at  CMM  Level  3.  In  2005  it  was 
using  the  results  of  a  SCAMPI  C  assessment  to  help  formulate  a  transition  from  CMM  to 
CMMI. 

F/A-18  SWDTT 

The  F/A-18  SWDTT  is  a  70  member  sub-team  of  the  F/A-18  Advanced  Weapons 
Lab  (AWL).  It  was  among  the  earliest  adopters  of  process  improvement  as  a  way  to 
reduce  cost  and  improve  quality.  F/A-18  SWDTT  has  pursued  SPI  since  the  early  1990s 
and  in  December  1997  achieved  a  CMM  Level  3  rating.  In  November  2000  there  was  a 
setback  when  the  SPI  lead  left  during  a  heavy  turnover  in  personnel  (104  personnel 
within  1(4  years).  When  the  SPI  Lead  position  was  filled  in  November  2001,  F/A-18 
SWDTT  realized  that  any  institutionalization  of  its  process  improvement  had  been  lost. 
The  decision  was  made  to  reorganize  and  restart  the  SPI  initiative.  F/A-18  SWDTT 
developed  a  plan  based  on  the  results  of  a  CBA  IPI  assessment  conducted  in  April  2003. 
As  part  of  that  plan,  TSP  was  selected  for  use  within  the  team  and  a  TSP  launch  was 
conducted.  F/A-18  SWDTT  improved  its  time  tracking  tools  and  established  a  web  site  to 
provide  information  and  resources  to  support  team  process  improvement.  F/A-18 
SWDTT  also  generated  a  “Five  Step  Model”  for  the  organization  (Reference  5): 

1 .  Focus  on  familiarization,  re-education,  and  training. 

•  Understand  that  these  need  to  be  continuous. 

•  Update  and  present  training/orientation  packages. 

2.  Recognize  process  compliance. 

•  Observe:  communicate  what  has  been  observed. 

•  Recognize:  brief  team  members  on  metrics  that  are  gathered  and  used. 

•  Make  it  cultural:  talk  about  it. 

3.  Complete  a  Self-Assessment  and  Gap  Analysis. 

•  “Health  check”  for  process. 

•  Next  steps. 

4.  Translate  the  process  changes  into  something  meaningful  to  the  engineering  staff. 

•  Previous  process  changes  had  left  the  engineering  staff  feeling  uninvolved. 

•  Time  saving. 

•  Effort  saving. 
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5.  Collect  and  use  your  metrics. 

•  Improve  cost  estimates. 

•  Improve  schedules. 

•  Improve  product  quality. 

•  “Steps”  must  be  concurrent. 

•  “Steps”  must  be  sustained. 

•  Process  Improvement  should  be  a  project. 

F/A-18  SWDTT  conducted  a  SCAMPI  B  in  October  2004  and  used  the  results  of 
that  assessment  to  prepare  for  a  SCAMPI  A.  The  effort  was  successful  and  in  March 
2005,  F/A-18  SWDTT  was  assessed  at  CMM  Level  5,  the  first  in  the  Navy. 

Following  the  Level  5  assessment,  F/A-18  SWDTT  set  new  goals  that  included 
assisting  all  the  appropriate  AWL  teams  in  the  entire  F/A-18  AWL  to  achieve  CMMI 
Level  3  maturity.  If  successful,  this  would  encompass  the  development,  enhancement, 
and  maintenance  of  over  10  million  SLOC. 


OVERALL 

Overall,  the  SSAs  made  steady  SPI  progress  and  delivered  significant  achievements 
(see  Figure  5).  It  is  important  to  note  again  the  rapid  progression  of  several  of  the  SSAs 
through  the  CMM  levels.  Using  TSP,  AV-8B  and  P-3C  were  able  to  achieve  CMM 
Level  4  in  less  than  3  years:  March  2000  to  September  2003  for  AV-8B,  and  February 
2002  to  May  2004  for  P-3C.  The  SEI  average  for  progressing  from  CMM  Level  1  to 
Level  4  is  5 'A  years  (Reference  2).  The  colored  zones  in  Figure  5  represent  recommended 
CMM  goals  that  were  set  in  the  February  1999  NAVAIR  Team  Software  Strategic  Plan. 
The  lines  connecting  the  milestones  are  there  only  to  group  certain  information. 
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Year 


Other  milestone  |  | _ 

First  TSP  Launch  A  A 

CMM  Level  Milestone  O  O 


□ 

A  A 

O  • 


O 


F/A-18  E-2  AV-8B  P-3C  EA-6B  TACAIR 

EW 


NAVAIR  1999  Targets 
for  CMM  Achievement 

CMM  Level  5 

CMM  Level  3 


FIGURE  5.  Time  Line  of  SAA  CMM  Progress  and  Related  Milestones. 


THE  FUTURE  OF  PROCESS  IMPROVEMENT  WITHIN  NAVAIR 


NAVAIR  4.0’s  SPI  efforts  have  been  successful  in  developing  mature  organizations 
and  in  obtaining  an  excellent  ROI.  The  early  SPI  adopters  are  meeting  their  missions, 
producing  higher  quality  products,  and  generating  significant  cost  savings.  Their  success 
stories  have  inspired  other  SSAs  within  NAVAIR  4.0  to  adopt  SPI.  Fifteen  of  the 
18  SSAs  that  were  not  among  the  early  adopters  of  SPI  are  now  pursuing  SPI  in  some 
form.  Figure  6  is  an  Earned  Value  Chart  generated  by  the  TSP  tool  of  the  Anti-Radiation 
Missile  (ARM)  UPC  0.4  Software  Development  Team  documenting  that  these  new 
adopters  are  experiencing  the  same  performance  improvement  as  the  early  adopters. 

SPIKE  Program  Manager  Mr.  Steven  D.  Felix  said  of  their  recent  PI  efforts  “.  .  . 
TSP  was  a  major  contributor  to  the  success  of  our  project.  Most  processes  assume  large 
teams  and  huge  budgets,  and  because  of  this  are  of  no  value  to  small  projects  like  SPIKE. 
Never  have  I  seen  a  process  that  was  so  scaleable  that  it  was  useful  to  a  team  as  small  as 
SPIKE.  The  metrics  collected  are  tuned  for  our  project.  When  a  metric  did  not  show  any 
value,  we  could  stop  collecting  it  and  figure  out  what  to  collect  that  did  make  sense.  TSP 
has  been  so  successful  that  the  SPIKE  project  has  adapted  it  to  our  hardware  design 
process.” 
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FIGURE  6.  Earned  Value  Chart  of  the  ARM  UPC  0.4  Software  Team. 


As  these  new  adopters  continue  to  make  progress  with  SPI,  the  recurring  savings 
will  allow  NAVAIR  to  redirect  even  more  funds  to  the  Fleet  for  procurement  of  critical 
needs,  including  new  aircraft. 

As  part  of  the  effort  to  promote  process  improvement  amongst  the  SSAs,  NAVAIR 
4.0  created  the  NAVAIR  Software/Systems  Support  Center  (NSSC),  an  organization 
tasked  with  providing  assistance  and  guidance  with  model-based  and  process-based 
performance  improvement.  In  pursuit  of  this  mission,  the  NSSC: 

•  Developed  an  internal  appraisal  method  based  on  the  SEI  Appraisal  Requirements 
for  CMMI  (ARC)  document  to  baseline  SPI  efforts  across  NAVAIR. 

•  Works  to  expand  the  number  of  SSAs  within  NAVAIR  4.0  that  are  pursuing  SPI. 

•  Sponsors  the  NAVAIR  Software  Process  Improvement  Community  of  Practice  (SPI 
CoP)  quarterly  conferences.  Representatives  from  all  NAVAIR  4.0  SEPGs  attend 
these  conferences.  SPI  CoPs  are  a  forum  for  exchanging  SPI  histories,  best  practices, 
techniques,  tools,  and  lessons  learned. 

•  Sponsors  the  NAVAIR  TSP  CoP  monthly  meetings.  TSP  coaches  and  instructors 
who  support  NAVAIR  4.0’s  TSP  teams  attend  these  meetings.  They  coordinate 
future  events,  share  best  practices,  and  keep  each  other  aware  of  the  status  of 
ongoing  efforts. 
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•  Works  with  the  SEI  to  introduce  and  pilot  TPI  projects,  a  modified  TSP  for 
acquisition  and  SE  process  improvement.  This  has  resulted  in  the  creation  of  new 
courses,  including  the  2-day  SE  focused  course,  Team  Member  Training. 

•  Provides  SEI-authorized  training  in  CMMI  and  coaching  in  PSP/TSP/TPI. 

The  charter  of  the  NSSC  also  calls  for  reinforcing  and  aligning  NAVAIR  4.0’s  SPI 
initiatives  with  NAV AIR’s  general  process  improvement  initiatives,  such  as  the 
AIRSpeed  lean  six-sigma  project.  While  the  efforts  of  NAVAIR  AIRSpeed  do  not  fall 
under  the  scope  of  this  document,  AIRSpeed  is  described  here  as  an  important  part  of  the 
overall  process  improvement  environment.  NAVAIR  AIRSpeed  lean  six-sigma  was 
selected  as  the  Naval  Aviation  Enterprise  (NAE)-wide  mechanism  for  reducing  costs  and 
improving  productivity  and  customer  satisfaction.  AIRSpeed  utilizes  a  structured, 
problem  solving  methodology  called  DMAIC  (Define,  Measure,  Analyze,  Improve, 
Control),  widely  used  in  business.  DMAIC  leads  project  teams  through  the  logical  steps 
from  problem  definition  to  problem  resolution.  Each  phase  has  a  specific  set  of 
requirements  to  be  met  before  moving  on  to  the  next  phase  of  the  project.  A  summary  of 
steps  are  as  follows: 

•  Define  the  problem. 

•  Measure  the  baseline  performance  of  the  process  being  transformed. 

•  Analyze  the  process  to  determine  a  prioritized  list  of  root  causes  for  any  process 
performance  shortfalls. 

•  Improve  the  target  process  by  designing  innovative  solutions  to  resolve  the  identified 
root  causes. 

•  Control  the  process  to  ensure  that  the  improved  process  continues  to  deliver  the 
expected  results. 

AIRSpeed  solicits  recommendations  on  process  change  from  all  NAVAIR  personnel 
and  contractors.  The  responsibility  for  following  up  on  those  recommendations  rests  with 
specially  trained  personnel  (Black  Belts). 

As  part  of  the  effort  to  institutionalize  lean  six-sigma,  NAVAIR  organized  and  held 
an  annual  NAVAIR  lean  six-sigma  symposium.  The  Navy,  looking  for  a  way  to  spread 
lean  six-sigma  throughout  the  organization,  saw  the  success  of  NAVAIR’ s  event  and  had 
NAVAIR  establish  the  first  Navy-wide,  lean  six-sigma  symposium  in  May  2007. 

Not  content  with  just  experiencing  the  tangible  advantages  of  process  improvement, 
NAVAIR  4.0  is  devoted  to  spreading  SPI  throughout  the  Navy,  the  Federal  Government, 
industry,  and  beyond.  NAVAIR  4.0  is  a  co-sponsor  of  DOD’s  Crosstalk  magazine;  has 
been  the  sponsor  of  various  conferences,  workshops,  and  panel  discussions;  and  has 
published  numerous  articles  on  SPI  (see  Appendix  E).  NAVAIR  4.0  personnel  participate 
in  the  international  TSP  Users  Group  (TUG)  conferences,  with  one  NAVAIR  employee 
holding  the  office  of  TUG  Conference  Chairman  for  2  years;  the  annual  National  Defense 
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Industrial  Association  (NDIA)  CMMI-User’s  Conferences;  and  the  annual  SEI  SEPG 
Conferences.  This  effort  on  behalf  of  SPI  has  been  noticed  by  the  wider  community, 
which  gave  a  NAVAIR  employee  the  2007  SEI  Member  Representative  award. 


CONCLUSIONS 


NAVAIR  4.0  had  recognized  the  need  for  and  was  pursuing  SPI  even  before  the 
U.S.  Government  entered  into  the  issue  with  Public  Law  107-314.  NAVAIR  4.0’s 
dedicated  interest  in  SPI  resulted  in: 

•  The  first  CMM  Level  5  rating  in  the  Navy. 

•  The  second  EVMS  certification  in  the  Federal  Government. 

•  Two  teams  achieving  CMM  Level  4  in  less  than  half  the  normal  time. 

•  Defect  density  reductions  ranging  from  22  to  48%. 

•  Cost  and  schedule  variances  reduced  to  within  10%. 

SPI  also  paid  impressive  ROIs.  AV-8B  and  P-3C  invested  a  combined  total  of 
$536,000  into  the  adoption  of  TSP  and  saw  a  combined  gross  savings  from  their  first  TSP 
efforts  (through  the  avoidance  of  rework)  of  an  estimated  $3,283  million.  This  gave  a  net 
ROI  of  approximately  $2,746  million.  EA-6B,  using  the  WDR  process  and  lean  six  sigma 
concepts,  saved  at  least  $1 16,000. 

The  successes  that  NAVAIR  4.0  has  enjoyed  from  SPI  and  its  culture  of  process 
improvement  have  helped  to  ensure  the  continued  pursuit  of  and  advancement  of  SPI 
within  the  organization.  A  belief  in  the  real  value  of  SPI  to  the  Navy  and  to  the 
Government  has  created  a  NAVAIR  4.0-wide  software  development  community  with  a 
desire  to  spread  SPI  outside  its  own  ranks.  Government  bureaucracies  are  wary  of 
change,  and  many  will  actively  resist  change  unless  they  are  provided  with  concrete 
examples  of  its  advantages.  This  resistance  often  begins  with  the  individual  workers  and 
extends  up  through  middle  management.  When  a  Government  organization  recognizes 
the  need  for  change,  and  the  personnel  throughout  that  organization  actively  seek  it  out,  a 
fundamental  shift  in  paradigms  has  taken  place.  NAVAIR  4.0  demonstrated  this  dramatic 
change  in  thinking  through  the  widespread  adoption  and  institutionalization  of  CMMI 
and  TSP.  It  is  an  operational  example  of  the  concrete  advantages  of  pursuing  SPI:  higher 
quality  products,  at  lower  cost,  while  maintaining  the  mission. 

In  January  2003,  Darrell  Maxwell,  at  that  time  Department  Head  of  the  NAVAIR 
Systems  Engineering  Department,  made  the  following  statement  after  reviewing 
NAVAIR  4.0’s  organization  and  the  strides  that  had  been  made  in  just  3 Vi  years  of  SPI 
efforts:  “In  February  1999  we  at  NAVAIR  set  out  to  make  notable  achievements  in 
software  process  improvement  across  the  organization.  It  was  just  good  business.  It  is 
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now  January  2003,  and  we  have  not  only  met  our  goals,  but  in  some  cases  achieved  even 
more  than  we  planned.” 


RECOMMENDATIONS 


A  complete  discussion  of  the  introduction  of  process  improvement  into  an 
organization  is  not  within  the  scope  of  this  document.  However  there  are  some  basic 
concepts  and  resources  that  will  help.  Process  improvement  can  be  a  difficult 
undertakings,  and  if  it  is  not  pursued  in  a  systematic  fashion,  it  will  be  much  more 
difficult.  SEI  identified  eight  key  concepts  for  introducing  process  improvement  into  an 
organization  ,  with  the  focus  on  introducing  CMMI.  The  concepts  are  summarized  here. 

1.  Secure  Sponsorship  and  Funding.  First,  ensure  that  the  process  improvement  effort 
has  both  a  senior  management  sponsor  and  funding. 

2.  Take  Core  Training.  Understand  the  basic  concepts  of  the  tools  and  methodologies 
that  will  be  used  in  the  process  improvement  effort. 

3.  Prepare  the  Organization  for  Change.  Treat  process  improvement  as  a  project. 
Establish  business  reasons  and  goals  for  the  effort.  Create  a  case  and  rationale  for 
undertaking  this  change.  Identify  the  expected  costs  and  benefits  for  everyone.  Plan 
for  and  manage  the  human  side  of  change. 

4.  Form  an  Engineering  Process  Group  (EPG).  The  EPG  should  coordinate  the  process 
improvement  activities  across  the  organization. 

5.  Know  Where  You  Are.  Survey  the  organization  and  compare  their  processes  to  those 
of  the  tools  and  methodologies  that  will  be  used  in  the  process  improvement  effort. 
This  will  serve  as  a  baseline  for  the  effort. 

6.  Know  Where  You  Are  Going.  Determine  the  overall  process  improvement  goals  for 
the  organization.  Prioritize  the  areas  to  be  addressed  and  then,  using  the  baseline  for 
the  organization  as  the  starting  place,  plan  the  path  to  achieve  those  goals.  Track  the 
organization’s  progress  against  the  plan. 

7.  Communicate  and  Coordinate.  Promote  and  practice  honest  and  open  communica¬ 
tion.  The  plan  should  be  shared  with  all  affected  parties.  Comments  and  concerns 
should  be  taken  seriously. 

8.  Track  Your  Progress.  Monitor  the  progress  of  the  organization  through  the  plan, 
making  adjustments  as  needed. 


"http://www.sei. cmu.edu/cmmi/adoption/cmmi-start.html,  2007. 
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RESOURCES 

Numerous  resources  are  available  both  inside  and  outside  of  the  Navy  to  assist 
process  improvement  efforts.  Following  is  a  partial  list  of  organizations  and  associations 
that  may  be  of  interest. 


Within  the  Navy 

NAVAIR  AIRSpeed.  The  NAVAIR  mechanism  for  reducing  costs  and  improving 
productivity  and  customer  satisfaction.  AIRSpeed  utilizes  the  DMAIC  (Define,  Measure, 
Analyze,  Improve,  Control)  structured  problem  solving  method.  The  official  AIRSpeed 
web  site  is 

http://www.navair.navy.mil/navairairspeed/. 

NAVAIR  Systems/Software  Support  Center.  A  resource  for  support  with  the 
introduction  and  maintenance  of  process  improvement.  NSSC  is  available  to  directly 
support  CMMI,  PSP,  TSP,  and  TPI.  NSSC  contact  for  assistance:  760-939-6226  (DSN 
437-6226).  The  home  code  for  this  organization  is  NAVAIR  Code  414300D. 

People,  Process,  and  Product  Resource  (P3R)  Group.  A  resource  for  personnel 
and  organizational  development  and  the  introduction  and  management  of  process 
improvement.  P3R  is  an  enterprise  team  sponsored  by  NAVAIR  Code  4.1. 

The  Department  of  the  Navy  Chief  Information  Officer  (DoN  CIO).  A  resource 
for  infonnation  and  guidance  on  the  requirements  associated  with  process  improvement 
initiatives  within  the  Navy  and  Federal  Govermnent,  as  well  as  links  to  resources  for 
process  improvement  tools  and  methodologies.  The  DoN  CIO  web  site  is 
http://www.doncio.navy.mil. 


Outside  the  Navy 

Software  Engineering  Institute,  Carnegie  Mellon  University.  SEI  is  a 

Government-funded  research  center.  Their  focus  is  on  software  process  improvement, 
security,  performance  measurement,  architecture,  acquisition,  and  other  important  aspects 
of  the  software  industry.  The  SEI  web  site  is  http://www.sei.cmu.edu/. 

Institute  of  Electrical  and  Electronics  Engineers  (IEEE).  IEEE,  an  international 
professional  association,  is  a  leader  in  the  advancement  of  technology.  The  IEEE  web 
site  is  http://www.ieee.org/portal/site. 

Armed  Forces  Communications  and  Electronics  Association  (AFCEA).  The 

focus  of  this  organization  is  Global  Security,  but  it  does  include  process  improvement 
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resources.  AFCEA  brings  together  professionals  from  Government  agencies,  industry, 
and  the  military,  providing  an  excellent  opportunity  to  network  with  other  professionals 
working  on  process  improvement.  The  AFCEA  web  site  is  http://www.afcea.org/. 

National  Defense  Industrial  Association.  NDIA  is  a  defense  industry  association 
and  the  industry  sponsor  of  CMMI.  It  advocates  advanced  technology  and  superior 
training  and  support  for  the  Anned  Forces  and  Emergency  Services.  The  NDIA  web  site 
is  http://ndia.org/. 
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Appendix  A 

NAVAIR  SOFTWARE  PROCESS  IMPROVEMENT  INSTRUCTIONS 


In  2002,  NAVAIR  issued  a  set  of  five  instructions  on  software  acquisition  and 
development. 


NAVAIRINST  5234.1 

Policy  on  Software  Evaluations  for  Naval  Air  Systems  Command,  dated 
30  September  2002.  The  goal  of  this  instruction  was  “To  Promulgate  policy  for  selecting 
software  prime  contractors  and  subcontractors  prior  to  awarding  software  intensive 
systems  acquisition  contracts  within  the  Naval  Air  Systems  Command.” 


NAVAIRINST  5234.2 

Requirements  for  Process  Improvement  Actions  for  Naval  Air  Systems  Command 
Software  Acquisition,  Development,  and  Life-Cycle  Support,  dated  29  May  2002.  The 
goal  of  this  instruction  was  “To  Promulgate  requirements,  responsibilities,  and  guidance 
for  improving  cost,  schedule,  and  technical  performance  of  Naval  Air  Systems  Command 
products  and  services  for  systems  software  acquisition,  development,  and  life  cycle 
support.”  The  instruction  was  based,  in  part,  on  the  research  of  Mr.  Jeff  Schwalb,  of  the 
Software  Resources  Team,  a  member  of  the  Business  Process  Reengineering  team 
(Reference  1,  page  9). 


NAVAIRINST  5234.3 

Establishment  of  the  System  Leadership  Council  and  the  Software  Leadership  Team, 
dated  17  April  2002.  The  goal  of  this  instruction  was  “To  establish  the  System 
Leadership  Council  and  the  Software  Leadership  Team  as  the  principals  for  leadership 
direction  of  software  systems  acquisition,  development,  and  maintenance  improvements 
within  the  Naval  Air  Systems  Command.” 


NAVAIRINST  5234.4 

Independent  Expert  Program  Reviews  for  Software  Intensive  Programs,  dated 
21  June  2002.  The  goal  of  this  instruction  was  “To  implement  the  requirements  of 
paragraph  C5.2.3.5.6.3  of  reference  (a)  and  to  provide  guidelines  for  conducting 
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Independent  Expert  Program  Reviews  (IEPRs)  within  the  Naval  Air  Systems  Command.” 
Reference  (a)  was  the  Department  of  Defense  Regulation  DOD  5000. 2-R,  Mandatory 
Procedures  for  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major  Automated 
Information  System  (MAIS)  Acquisition  Programs.’’'’ 


NAVAIRINST  5234.5 

Naval  Air  Systems  Command  Metrics  for  Software  Intensive  Programs,  dated 
30  September  2002.  The  goal  of  this  instruction  was  “To  establish  a  basic  set  of  software 
metrics  for  managing  performance  (financial,  customer  value,  and  quality)  in  acquiring, 
developing,  and  maintaining  software  intensive  systems.” 
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Appendix  B 

NAVAIR  ORGANIZATIONAL  GOALS 


In  2005,  NAV AIR’s  Organizational  Goals  were  stated  on  the  NAVAIR  web  site. 

1.  To  Balance  Current  and  Future  Readiness.  We  need  to  ensure  that  we  provide  our 
Naval  aviators  with  the  right  products  to  fight  the  Global  War  on  Terrorism  and 
other  future  conflicts. 

2.  To  Reduce  Our  Costs  of  Doing  Business.  We  need  to  pursue  actual  cost  reductions, 
not  so-called  “savings”  or  “avoidance.”  We  need  to  return  resources  to  recapitalize 
our  Fleet  for  the  future.  We  must  continue  to  introduce  best  business  practices  and  to 
remove  any  barriers  to  getting  our  job  done. 

3.  To  Improve  Agility.  It  is  essential  that  we  make  rapid  decisions  in  support  of 
emerging  Fleet  requirements  in  order  to  continue  to  provide  value  to  the  nation.  We 
must  reinvigorate  a  solid  chain  of  command  that  values  responsibility  and 
accountability  in  its  leadership. 

4.  To  Ensure  Alignment.  We  have  come  a  long  way  with  aligning  ourselves  internally. 
Now  we  must  ensure  that  we  are  fully  aligned,  internally  and  externally,  with  the 
Chief  of  Naval  Operation’s  (CNO)  transfonnation  initiatives. 

5.  To  Implement  Fleet-Driven  Metrics.  Single  Fleet-driven  metrics  will  ensure  that  we 
directly  contribute  to  the  Naval  Aviation  Enterprise. 
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Appendix  C 

DOD  CRITERIA  FOR  EARNED  VALUE  MANAGEMENT  SYSTEM 

CERTIFICATION 


ORGANIZATION 

•  Define  contract  work  using  work  breakdown  structure. 

•  Identify  organizational  responsibilities  to  include  sub-contractors. 

•  Integrate  planning,  scheduling,  budgeting,  work  authorization,  and  cost  accumula¬ 
tion. 

•  Identify  overhead  control  responsibilities. 

•  Measure  performance  by  Work  Breakdown  Structure  (WBS)  and  organizational 
breakdown. 


PLANNING  AND  BUDGETING 

•  Schedule  work  showing  task  inter-dependencies. 

•  Identify  physical  products,  milestones,  and  technical  performance  progress  metrics. 

•  Establish  and  maintain  a  performance  measurement  baseline. 

•  Establish  work  budgets. 

•  Establish  work  and  planning  packages. 

•  Reconcile  all  work/planning  package  budgets  within  a  control  account  with  that 
control  account’s  budget. 

•  Identify  and  control  the  level  of  effort  (LOE). 

•  Identify  overhead  budgets. 

•  Identify  Management  Reserves  (MR)  and  undistributed  budgets. 

•  Reconcile  the  project  cost  goal  with  internal  budgets  and  MR. 


ACCOUNTING 

•  Record  direct  costs  consistent  with  work  budgets. 

•  Summarize  direct  costs  without  allocation  to  two  or  more  WBSs. 

•  Summarize  direct  costs  without  allocation  to  two  or  more  organization  elements. 

•  Record  all  indirect  costs. 

•  Identify  unit/equivalent  unit  or  lot  costs,  when  needed. 

•  Provide  full  accountability,  performance  measurement,  and  accurate  cost  accumula¬ 
tion. 
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ANALYSIS  AND  MANAGEMENT  REPORTS 

•  At  least  monthly,  provide  management  with  information  on  planned/accomplished 
work  and  costs. 

•  At  least  monthly,  identify  direct  cost/schedule  variances. 

•  Identify  indirect  cost  variances  as  needed. 

•  Summarize  variances  by  WBS  and/or  organizational  element. 

•  Implement  actions  based  upon  earned  value  (EV)  information. 

•  Develop  estimates  of  costs  at  completion. 


REVISIONS  AND  DATA  MAINTENANCE 

•  Timely  incorporate  changes. 

•  Control  internal  re -planning. 

•  Control  retroactive  changes. 

•  Change  the  budget  only  when  authorized. 

•  Document  changes  to  the  performance  measurement  baseline. 
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Appendix  D 

STANDARD  CMMI  ASSESSMENT  METHOD 
FOR  PROCESS  IMPROVEMENT 


SCAMPI  A 

The  following  description  of  the  purpose  and  benefits  of  a  SCAMPI  A  is  quoted 
from  Standard  CMMIsm  Appraisal  Method  for  Process  Improvement  (SCAMPIsm), 
Version  1.1:  Method  Definition  Document  (Reference  6,  pages  1-6). 

“The  Standard  CMMI  Appraisal  Method  for  Process  Improvement 
(SCAMPI)  is  designed  to  provide  benchmark  quality  ratings  relative  to 
Capability  Maturity  Model  Integration  (CMMI)  models.  It  is  applicable  to  a 
wide  range  of  appraisal  usage  modes,  including  both  internal  process 
improvement  and  external  capability  determinations.  SCAMPI  satisfies  all  of 
the  Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a  Class  A 
appraisal  method  and  can  support  the  conduct  of  ISO/IEC  [International 
Organization  for  Standardization  /  International  Electrotechnical  Commission] 
15504  assessments.” 

“SCAMPI  V 1 . 1  enables  a  sponsor  to 

•  gain  insight  into  an  organization’s  engineering  capability  by  identifying  the 
strengths  and  weaknesses  of  its  current  processes 

•  relate  these  strengths  and  weaknesses  to  the  CMMI  model 

•  prioritize  improvement  plans 

•  focus  on  improvements  (correct  weaknesses  that  generate  risks)  that  are 
most  beneficial  to  the  organization  given  its  current  level  of  organizational 
maturity  or  process  capabilities 

•  derive  capability  level  ratings  as  well  as  a  maturity  level  rating 

•  identify  development/acquisition  risks  relative  to  capability/maturity 
determinations” 

“As  a  Class  A  appraisal  method,  SCAMPI  is  an  appropriate  tool  for 
benchmarking.  Sponsors  who  want  to  compare  an  organization’s  process 
improvement  achievements  with  other  organizations  in  the  industry  may  have  a 
maturity  level  determined  as  part  of  the  appraisal  process.” 
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Types  of  SCAMPIs 

The  following  general  description  of  the  three  types  of  SCAMPIs  was  taken  from 
Handbook  for  Conducting  Standard  CMMI  Appraisal  Method  for  Process  Improvement 
(SCAMPI)  B  and  C  Appraisals,  Version  1.1  (Reference  7,  page  4).  Only  SCAMPI  A  will 
result  in  a  fonnal  CMMI  rating.  SCAMPI  B  and  C  are  used  to  guide  an  organization 
along  the  road  towards  preparing  to  conduct  SCAMPI  A. 

“The  SCAMPI  family  architecture  differentiates  three  classes  of  methods 
by  identifying  the  primary  focus  of  SCAMPI  A,  B,  and  C  as 
“institutionalization,”  “deployment,”  and  “approach,”  respectively.  The 
SCAMPI  A  method  has  rigorous  standards  for  detailed  data  collection,  and  for 
identification  and  coverage  of  the  organizational  unit.  The  SCAMPI  B  method 
retains  some  of  the  requirements  for  detailed  data  collection,  but  provides 
relaxed  standards  for  sampling  the  organization.  The  SCAMPI  C  method  has 
relaxed  standards  relating  to  evidence  of  usage.  These  methods  can  form 
building  blocks  for  a  progression  of  appraisals-for  example,  starting  with  a 
SCAMPI  C  reviewing  the  process  descriptions,  then  a  SCAMPI  B  investigating 
their  deployment  to  projects,  finally  leading  to  a  formal  benchmarking  event 
focused  on  institutionalization  of  the  practices  across  the  organization.” 
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Appendix  E 

CODE  4.0  ARTICLES  AND  PRESENTATIONS  RELATED  TO  SOFTWARE 

PROCESS  IMPROVEMENT 


1.  “AV-8B’s  Experience  Using  the  TSP  to  Accelerate  SW-CMM  Adoption,”  Crosstalk, 
September  2002 

2.  “AV-8B  JSSA  Team  Soars  to  Level  4,”  NAVAIR  press  release,  January  2003. 

3.  “Team  Software  Process  for  Maintenance  Projects,”  Software  Engineering  Process 
Group  Conference,  Boston,  Massachusetts,  February  2003. 

4.  “Accelerating  SW-CMM  Progress  Using  the  TSP,”  Software  Engineering  Process 
Group  Conference,  Boston,  Massachusetts,  February  2003. 

5.  “CMM®  +  PSP/TSPsm  — >  CMMI®,”  Software  Technology  Conference,  Salt  Lake 
City,  Utah,  April  2003. 

6.  “Team  Software  Process  for  Maintenance  Projects,”  Software  Technology  Confer¬ 
ence,  Salt  Lake  City,  Utah,  April  2003. 

7.  “Team  Software  Process:  A  Practitioner’s  Checklist,”  TSP  User’s  Group  Confer¬ 
ence,  Pittsburgh,  Pennsylvania,  September  2003. 

8.  “Team  Software  Process  for  Maintenance  Projects,”  TSP  User’s  Group  Conference, 
Pittsburgh,  Pennsylvania,  September  2003. 

9.  “The  AV-8B  Team  Learns  Synergy  of  EVM  and  TSP  Accelerates  Software  Process 
Improvement,”  Crosstalk,  January  2004. 

10.  “AVJMPS  Project  Summary  and  Lessons  Learned,”  Software  Engineering  Process 
Group  Conference,  Orlando,  Florida,  March  2004. 

11.  “A  Life-Cycle  for  Corrective  Maintenance  TSP  Projects,”  Software  Engineering 
Process  Group  Conference,  Orlando,  Florida,  March  2004. 

12.  “Team  Software  Process:  A  Practitioner’s  Checklist,”  System/Software  Technology 
Conference,  Salt  Lake  City,  Utah,  April  2004. 

13.  “AVJMPS  Project  Summary  and  Lessons  Learned,”  System/Software  Technology 
Conference,  Salt  Lake  City,  Utah,  April  2004. 
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14.  “Tools  And  Techniques  For  Making  Your  TSP  Team  More  Effective,”  TSP  User’s 
Group  Conference,  Pittsburgh,  Pennsylvania,  September  2004. 

15.  “Lessons  Learned,”  SEPG  User ’s  Conference,  Seattle,  Washington,  March  2005. 

16.  “A  TSP  Software  Maintenance  Life  Cycle,”  Crosstalk,  March  2005. 

17.  “Process  Improvement  at  NAVAIR  using  Capability  Maturity  Models  and  TSP,” 
Software  Engineering  Process  Group  Conference,  Nashville,  Tennessee,  March 
2006. 

18.  “Process  Improvement  at  NAVAIR  using  TSP  and  CMM,”  Team  Software  Process 
Symposium,  San  Diego,  California,  September  2006. 

19.  “NAVAIR  Lamp  Model  -  A  Coach’s  Aid  in  Helping  Teams  Apply  TSP,”  Team 
Software  Process  Symposium,  San  Diego,  California,  September  2006. 

20.  “Interdisciplinary  Team  Project  Management  Using  TSP  Concepts,”  Team  Software 
Process  Symposium,  San  Diego,  California,  September  2006;  and  NDIA  Systems 
Engineering  Conference,  San  Diego,  California,  October  2006. 

21.  “NAVAIR  TSP  Experience  Using  the  Excel  Workbook  and  Team  Dashboard,” 
Systems  and  Software  Technology  Conference,  Salt  Lake  City,  Utah,  May  2006;  and 
Team  Software  Process  Symposium,  San  Diego,  California,  September  2006. 

22.  “How  Applying  the  Team  Software  Process  and  Personal  Software  Process  Can 
Increase  Software  Supportability,”  NDIA  Systems  Engineering  Conference,  San 
Diego,  California,  October  2006. 
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ACRONYMS 


AEA  Airborne  Electronic  Attack 

AFCEA  Armed  Forces  Communications  and  Electronics  Association 
ARBS  Angle  Rate  Bombing  System 

ARC  Appraisal  Requirements  for  CMMI 

ARM  Anti-Radiation  Missile 

AVJMPS  AV-8B  Joint  Mission  Planning  System 

AWL  Advanced  Weapons  Lab 

BLK  Block 

CBA  IPI  CMM-Based  Appraisal  for  Internal  Process  Improvement 
CCHPO  Commonwealth  Centers  for  High  Perfonnance  Organizations 

CM  Configuration  Management 

CMMI  Capability  Maturity  Model  Integration 

CNO  Chief  of  Naval  Operations 

DMAIC  Define,  Measure,  Analyze,  Improve,  Control 

DOD  Department  of  Defense 

DoN  CIO  Department  of  the  Navy  Chief  Information  Officer 

EPG  Engineering  Process  Group 

EV  Earned  Value 

EVMS  Earned  Value  Management  System 

EW  Electronic  Warfare 

GWT  Global  War  on  Terrorism 

HPO  High  Perfonnance  Organization 

ICAP  Initial  Capability 

IEC  International  Electrotechnical  Commission 

IEEE  Institute  of  Electrical  and  Electronics  Engineers 
IEPR  Independent  Expert  Program  Review 

IOC  Initial  Operational  Capability 

ISO  International  Organization  for  Standardization 

JSF  Joint  Strike  Fighter 

JSSA  Joint  System  Support  Activity 

KSLOC  One  thousand  source  lines  of  code 


G-l 


NAWCWD  TP  8642 


LOE  Level  of  Effort 

MAIS  Major  Automated  Information  System 

MDAP  [U.S.  DOD]  Major  Defense  Acquisition  Program 

MR  Management  Reserves 

MSC  Mission  Systems  Computer 

NAE  Naval  Aviation  Enterprise 

NAVAIR  Naval  Air  Systems  Command 

NCW  Network-Centric  Warfare 

NDIA  National  Defense  Industrial  Association 

NSSC  NAVAIR  Software/Systems  Support  Center 
NTS  Night  Targeting  System 

OFP  Operational  Flight  Program 

OTRR  Operational  Test  Readiness  Review 

P3R  People,  Process,  and  Product  Resource  [group] 

PI  Process  Improvement 

PSP  Personal  Software  Process 

ROI  Return  on  Investment 

RUG  Radar  Upgrade 

SCAMPI  Standard  CMMI  Assessment  Method  for  Process  Improvement 
SE  System  Engineering 

SEI  [Carnegie  Mellon  University]  Software  Engineering  Institute,  Pitts¬ 
burg,  Pennsylvania 

SEPG  Software  Engineering  Process  Group 

SLOC  Source  Lines  of  Code 

SMUG  Stores  Management  Upgrade 

SPI  Software  Process  Improvement 

SPI  CoP  Software  Process  Improvement  Community  of  Practice 
SPR  Software  Problem  Report 

SSA  Software  Support  Activity 

STR  System  Trouble  Report 

S/W  Software 

SW-CMM  Software-CMM 

SWDTT  Software  Development  Task  Team 

SWIP  Software  Improvement  Program 

T  AC  AIR  Tactical  Aircraft 
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TPI 
TSP  CoP 
TSPm 
TUG 

UAV 

UPC 

V&V 

WBS 

WD 

WDR 

WSSL 


Team  Process  Integration 
Team  Software  Process  Community  of  Practice 
Team  Software  Process  for  Multiple  Teams 
TSP  Users  Group 

Unmanned  Aerial  Vehicle 
Unique  Planning  Component 

Verification  and  Validation 

Work  Breakdown  Structure 
Weapons  Division 
WSSL  Discrepancy  Reporting 
Weapon  System  Support  Laboratory 
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